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I shall camment briefly on four problems of data suinmary and anaiysis. 
The construction md analysis of the concept attainniint tests presented 
two problems. ¥or each subject matter, a set Q.f 56O items txiata in a completely 
, arossed design; the items form a matrix of, say twelve rows representing the 
^ twelve tas'<s and thirty columns representing the thirty coneepts, for 'a given 
individual who responds to all 36O items, this matrix can" be nsed to represent 
^his performanGe by filling in each cell with a "0" for an incorrect and a "1" 
for a' correct responsa* Tne probltm of how to use such student response data 
to improve itens or to aelact the better items arisas; this problem is not 
independent of the problem of how to score such a set of items, ' 

One method of dealing with the 3q0 possible reaponsea would be to 
aaiume that all responies indicate the gam# ac.iievement and simply sMple, 
at random, twenty or thirty of these responsee and add the scores to yield a 
single score describing concept at'tainmtnt for that student in that aubject 
matter* This would permit forming indefinitely many randomly parallel tests 
that could be' administered to different individu^s and/or to the same individual 
at different times , However, such a procedufe rest a on the aasiMptlon that the 
row headings and the column headings do not designate importast functional 
differences. This is an assumption we wished to test, and bo we used redundant 
sets of scores* For concepts, thirty scores were develop^^d by summing the 
columns of the task-cQaeept matrix, and, for tasks, twelve scores were daveloped 
by summing the rows. We recognise that thea# two sets of scores are not completely 
independent* Also, only in the aissence of concept-task interactlona do the two 
sets of scores reflfct the total vwiance in the matrix* This redundMt scoring 
was employed in conventional item ^alyais to yield, for each item, two criterion 
scores* We found no sifl^ple Md satisfaGtory way to. put the two sets, of item 
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parasiter estimates together, and so we examined them sepiirately, Wa feel 
that probleLis of _scoriiig and of item analysis for iteas that exist in a cospletely 
crossed rtesign are wortliy of further invtstigation and study. Certainly we did 
cat resolve apparent iSEues or solve these problems. It may be that test 
developers will discover fflore and more occasions on Which the complete crosaing 
of content and task is appropriatei if fiOj these problems of scoring and item 
analysis will deraan solution. 

A second problem associated with the meaaureiatnt of concept attainment 
waa that of dtttrmining the factorial structure of such item sets. It was clear 
that for any one subject matter, a three-mode analyeia was needed to. describe 
the task difflensions, the coneept dimtiisione, and the concept^taak interactions* 
However I we found the current prograffis limit td in two reapicts# ^or one, most 
work haa , been dome on a .companent-typt anslysis, as eoatrasted with a factor-, 
type analysis. We would like to see more oapability for the latter* Secondi 
there are, currently * rather sharp limits on the product of the number of 
Inatances in each of the two modes other than indiyiduale* We reduced this 
product in two wayg, by summing acroae the ten concapta within a eub*area and 
by summing acroai tasks. Both of thest Malyses tendtd to indicate no gubst^tial 
concept-taek interactions for any of the four subject matters* We recognize 
that the methods we were able to use here gave only pwtial answers to the 
question of eontent^tajik interaction, since in both analyses headings on one 
of . the dimtncions were aggregated, . 

We alao performed conventional factor analyses for the two seta of 
acorea for each of the fot;^' subject matter fields for both the 1970 and the 
1971 data* Our intent waa to apply the st rat for factor intarpretation 
that has be^ described else secured 
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3 initial and 6 derived aolutione for each matrix. The derived solutions tended 
to diffar ir.arkedly in the nwnber of coismon factors, with Alpha yielding the 
smallest number (usually one coMon factor) and Harris R-S^ the largest number 
with one €:<ception. This wide discrepancy made the application of our factor 
inttrpreti strategy difficult* Apparently our strategy^ which involves 

identifyins comparable comiaQn factors, is most appropriate when the derived 
solutions tend to have approximately the saae number of major common factors. 
This was true for the anaiySis of the combined concept attainment measures |nd 
of the cognitive abilities tests ^ and for these analyses the str^teET worKed 
well. But within a subject matter field, the disparity forcad us to consider 
whether or not the solutions with several factors were overdif ferentiating. 
Such a condition is gusgest#d when the oblique solutions have very high factor 
intercorrelatlans. This was fomd to be truei and we conoluded that a sijigle 
common factor oharacteri^inj thi achieyemtnt scores of a subject matter was the 
inost defensible interpretation* ^his interpretation was found to be consistent 
^ , . with the later results in which achievement measwes for the four 'subject matter 
fields were analysed together* 

A third problem was that of reducing the battti^ of 56 cognitive abilities 
teste administered in 1970 to approximattly 50 tests, without losing major factors 
that had been identlfitd* aenerally, one looks at the factor matriXs tries to 
find two variables that arc af complexity one for each factor mud chooses those 
to retain. Often— especially for orthogonal derived solutions-*it is not possible 
to identify two VM^iahlts that are tssentially of complexity one for a given 
factor, and it then ie necessary to "compromise," Our decisions were made pri- 
marily on the baaia of the eoaparable common fact a variabis were a 
possible candidate for inelualoni we relied heavily on the regression coefficients 
^ipattera matrix) of the oblique solutions in deeidiag* whether or not to choose 
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regression coefficients ^^go ths othsr wa^'' so to speak; that is, these pattern 
valuea are regresaioa coel'iicients for eaticating variables from factors i and 
not vice versa* 

If we regard thia problem of reducing a. battery in such a w^y as to 
retain each of tne major factors as a problem simile to that of estimating 
factor scores, then one night choose from several methods of .developing the. 
weights on the variables to use in estimating factors or factor scores. See 
Harris (I967) and Tucker (1971) for relevant analyses* Presumably those variables 
with aubstantial weights for estimating a given factor would be the ones from 
which to aelect-^providing of ooiirse that the variable has only trivial weights 
for estiraating any other factor. A first problem is that of selecting the eystem 
of developing the weights. We were influenced by Tucker's analysia that shows 
that if the factors are to be related to variables outside the set of variables 
on which the factor analysis was done, then the "idtal variable** or "least 
squares" weighting rpocedure is a good one. If we accept this, then oblique 
solutions which have been deriYad using the H^ris-Kaiser independent cluster 
iQ^gorithm have pattern matrices which are proportional to the weighta for ' 
estimating factors* Harris (1971) has eoflifflented on this elsewhere in connection 
with work done by Brogden (I969)* The conclusion ia to use this independent 
cluster model solution pattern matrix as the relevant matrix to examine and 
to choose Y^iables which are of complexity one in that matrix. Oiu- problem 
waa not solved that easily, however, since we had found the indtpemdent cluster 
algorithm gave a certain aiaowt of biporu*ity in the pattern matrix, and wa had 
eubstituted the Harrla-Kaiser A*A proportional to L aolutlon« In using the patterii 
matrix for this type of solution as a guide to the ohoict of variables to retain 
in tht see ond study I we in efftct assvmed that the inverse of A' A was weil 
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approximated by a diagonal matrix. It should be e.-nphasized that this probleai 
of how to reduce a battery and yet retain the factors th^f have been ideatified 
is not a trivial one. ■ 

The fourth problem I shall mention is that of identifying relationahips 
between the battery of concept attaimr.ent measures and the battery of cognitive 
abilities tests, Mfferent attitudes may be adopted in plaunins a study of the 
relations between two batteries j we adopted an interbattery approach. In 
contrast, some workers do not make a distinction between the two batteries 
and others approach the problem either by way of canonical correlation and 
canonical variates when a component -type solution is of interest, or by way 
of projectins the variables of the "dependent" battery into the common factor 
space of the "independent" battery when a factor- type solution is wanted. 

We used canonical correlation theory for two purposes: to derive 
the squared multiple correlations for eBtimatiag each variable in one battery 
£ro» the set of variables in the other batte^, and to determine the nufflber of 
sigiiificant c'jaaonical oorrelations to give us a guide to the number of inter- 
batteiy factors to be extracted. At this ooncluding stage of the project, 
we wished to be certain .that our generalizatioas about the dimensionality, of the 
relations between concept attaiafflent and cognitive abilities would reflect 
•coacera about the number of degrees of freedon available to ue. 

Our e3^erience indicated that the Tucker interbatteiyproee 

..... ^ 

(Tucker, 1958) could readily be adapted to our type of study. The adaptations 
consisted of using a different test for the awaber of factors, securing derived 
orthogonal aolutioas for the two sets of interbattery factors that employed 
the same orthonormal trans format ion » andjapplyiag the Harris-Kaiser procedure 
for securiag oblique solutions to the iaterbattery factors, ae three adaptations 



seemed to work well and may be considered by others. In general found that 
the inttrbattery procedure is especially useful in that audh factora art easy 
to extract and are easy to put into readily interpretable for=. Our reoonunendatioa 
. would be that if the derived solutions are to be secured by applying the sa^ne 
orthonormal solution (or the fia^e oblique transformation), the transformation 
matrix should be developed from both sets of factors simultaneously. This seems 
likely to give more interpretable results, especially when there is a substantial 
nuniber of variables in the smalltr battery. We are speaking here,, of course, 
of studies such as ours in which the two batteries are conceptually different, 
rahter than being interchangable, 'and the interest is in linking the two. ' ' 

our experience also indicated that the interbattery procedure gave ' 
more useful results than a conventional single solution involving both batteries 
aimultanaously. Both the relative number of .variables and the general If el of 
. correlatioas in the two batteries may influence the single solutionj the 
interbattery solution is not influenced by the in^aba^er^ correlation level, 
and may be relatively independent of the split of the variables into the two 



batteries. 



We also developed the squared multiple correlations associated with 
astifflatiag variables of one battery fr on the interbattery factor scores, 
(These factor ecoree are ImpUcit in th^ theei^, but are not miquely calculable. 
The estimation procedure, however , does not depend on actually having these ' 
fa.;tor scores available.) We believe that these squared multiple correlationa 
•hould MVS relativtly little shrinkage in a new stu^ from a similar population 
of students alnre we used the si^lflcance test to tstablish a (possibly) 
conservative number of dimensions for ©atimating the.oross-correlations. If 
''^v squared multipla correU^ 
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